Papers with transformer modules
Dynamic Routing Transformer Network for Multimodal Sarcasm Detection (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for multimodal sarcasm detection rely on fixed architectures to capture cross-modal incongruity. |
| Approach: | They propose a method that uses dynamic paths to activate different routing transformer modules with hierarchical co-attention adapting to cross-modal incongruity. |
| Outcome: | The proposed method is compared to state-of-the-art methods on a public dataset. |
An Empirical Study on the Transferability of Transformer Modules in Parameter-efficient Fine-tuning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning is a computationally expensive process . introducing new parameters to an already-large model can be considered a drawback. |
| Approach: | They investigate the capability of different transformer modules in transferring knowledge from a pre-trained model to a downstream task. |
| Outcome: | The proposed methods show that each transformer module is a winning ticket . they show that with only 0.003% updateable parameters, they can show acceptable performance on target tasks. |